102 research outputs found
Emergence of Compositional Representations in Restricted Boltzmann Machines
Extracting automatically the complex set of features composing real
high-dimensional data is crucial for achieving high performance in
machine--learning tasks. Restricted Boltzmann Machines (RBM) are empirically
known to be efficient for this purpose, and to be able to generate distributed
and graded representations of the data. We characterize the structural
conditions (sparsity of the weights, low effective temperature, nonlinearities
in the activation functions of hidden units, and adaptation of fields
maintaining the activity in the visible layer) allowing RBM to operate in such
a compositional phase. Evidence is provided by the replica analysis of an
adequate statistical ensemble of random RBMs and by RBM trained on the
handwritten digits dataset MNIST.Comment: Supplementary material available at the authors' webpag
Estimating the principal components of correlation matrices from all their empirical eigenvectors
We consider the problem of estimating the principal components of a
population correlation matrix from a limited number of measurement data. Using
a combination of random matrix and information-theoretic tools, we show that
all the eigenmodes of the sample correlation matrices are informative, and not
only the top ones. We show how this information can be exploited when prior
information about the principal component, such as whether it is localized or
not, is available by mapping the estimation problem onto the search for the
ground state of a spin-glass-like effective Hamiltonian encoding the prior.
Results are illustrated numerically on the spiked covariance model.Comment: 6 pages, 6 figures, to appear in Europhysics Letter
Criticality and Universality in the Unit-Propagation Search Rule
The probability Psuccess(alpha, N) that stochastic greedy algorithms
successfully solve the random SATisfiability problem is studied as a function
of the ratio alpha of constraints per variable and the number N of variables.
These algorithms assign variables according to the unit-propagation (UP) rule
in presence of constraints involving a unique variable (1-clauses), to some
heuristic (H) prescription otherwise. In the infinite N limit, Psuccess
vanishes at some critical ratio alpha\_H which depends on the heuristic H. We
show that the critical behaviour is determined by the UP rule only. In the case
where only constraints with 2 and 3 variables are present, we give the phase
diagram and identify two universality classes: the power law class, where
Psuccess[alpha\_H (1+epsilon N^{-1/3}), N] ~ A(epsilon)/N^gamma; the stretched
exponential class, where Psuccess[alpha\_H (1+epsilon N^{-1/3}), N] ~
exp[-N^{1/6} Phi(epsilon)]. Which class is selected depends on the
characteristic parameters of input data. The critical exponent gamma is
universal and calculated; the scaling functions A and Phi weakly depend on the
heuristic H and are obtained from the solutions of reaction-diffusion equations
for 1-clauses. Computation of some non-universal corrections allows us to match
numerical results with good precision. The critical behaviour for constraints
with >3 variables is given. Our results are interpreted in terms of dynamical
graph percolation and we argue that they should apply to more general
situations where UP is used.Comment: 30 pages, 13 figure
On the trajectories and performance of Infotaxis, an information-based greedy search algorithm
We present a continuous-space version of Infotaxis, a search algorithm where
a searcher greedily moves to maximize the gain in information about the
position of the target to be found. Using a combination of analytical and
numerical tools we study the nature of the trajectories in two and three
dimensions. The probability that the search is successful and the running time
of the search are estimated. A possible extension to non-greedy search is
suggested.Comment: 6 pages, 7 figures, accepted for publication in EP
Information content in continuous attractor neural networks is preserved in the presence of moderate disordered background connectivity
Continuous attractor neural networks (CANN) form an appealing conceptual
model for the storage of information in the brain. However a drawback of CANN
is that they require finely tuned interactions. We here study the effect of
quenched noise in the interactions on the coding of positional information
within CANN. Using the replica method we compute the Fisher information for a
network with position-dependent input and recurrent connections composed of a
short-range (in space) and a disordered component. We find that the loss in
positional information is small for not too large disorder strength, indicating
that CANN have a regime in which the advantageous effects of local connectivity
on information storage outweigh the detrimental ones. Furthermore, a
substantial part of this information can be extracted with a simple linear
readout.Comment: 20 pages, 6 figure
- …